Measuring Operating System Robustness

Robustness is becoming more important as critical software increasingly affects our daily lives. Success in building robust software requires understanding and improving the robustness of the operating system API, but to date there has been no accurate, reproducible way to measure robustness. This paper presents the first full-scale, quantitative measurements of operating system robustness. Each of 15 different operating system’s robustness is measured by automatically testing up to 233 POSIX functions and system calls with exceptional parameter values. The work identifies repeatable ways to crash operating systems with a single call, ways to cause task hangs within OS code, ways to cause task core dumps within OS code, failures to implement defined POSIX functionality for unusual conditions, and false indications of successful completion in response to exceptional input parameter values. Overall, only 55% to 76% of tests performed were handled robustly, depending on the operating system being tested. Approximately 6% to 19% of tests failed to generate any indication of error in the presence of exceptional inputs. Approximately 1% to 3% of calls tested failed to implement defined POSIX functionality for unusual, but specified, conditions. Between 18% and 33% of calls tested dumped core from within a POSIX function or system call, and five operating systems were completely crashed by individual user mode system calls with exceptional parameter values. The most prevalent sources of robustness failures were illegal pointer values, numeric overflows, and end-of-file overruns. The results indicate that there is significant opportunity for increasing robustness within current operating systems. However, the role of signals vs. error return codes is both controversial and the source of divergent implementation philosophies, forming a potential barrier to writing portable, robust applications. Acknowledgments: This research was sponsored by DARPA contract DABT63-96-C-0064.